Abstract

Using GEPD data from Peru, this paper examines the psychometric properties of the 4th grade student assessment module. We produce summary statistics of the fraction correct overall, by subdomain, and at the item level. We examine the test reliability of our domains. We then plot estimated probabilities of a correct responses on our six math items that are aligned with at least a 4th grade curriculum on the number of items correct to get an appropriate cutscore.

Future work will do a similar exercise with literacy.

## Joining, by = "codigo.modular"

Summary Statistics of Data

Before estimating the model, we begin by examining summary statistics and missing values in our item response data. To begin, we will show summary statistics of the fraction correct on the literacy, and math sections of the examination, as well as the fraction correct on broad sub-domains.

The sub-domains for the literacy section are:

The sub-domains for the math section are:

## Joining, by = "name"
Table 1: Summary Statistics of Fraction Correct on assess_4th_grade Assessment Domains for 1st Graders
Item Label Mean Std Dev Min 25th Percentile Median 75th Percentile Max # Complete Cases # Missing Cases Histogram
Overall
student_knowledge Fraction correct overall 76.94 16.7 -6.67 70 80 88.33 100 2681 0 ▁▁▁▁▂▆▇▅
Literacy
literacy_student_knowledge Fraction correct Literacy 89.49 16.53 -14.72 83.33 100 100 100 2681 0 ▁▁▁▁▁▁▃▇
letters Letter Identification 0.93 0.21 -1 1 1 1 1 2681 0 ▁▁▁▁▁▁▁▇
words Word Recognition 0.97 0.13 -0.17 1 1 1 1 2681 0 ▁▁▁▁▁▁▁▇
comprehension Reading Comprehension Story 0.84 0.26 0 0.67 1 1 1 2681 0 ▁▁▁▁▁▃▁▇
Math
math_student_knowledge Fraction correct Numeracy 64.39 22.14 -2.22 51.11 66.67 80 100 2681 0 ▁▂▂▆▇▇▆▃
number_sense Number Sense 0.88 0.22 -0.2 1 1 1 1 2681 0 ▁▁▁▁▂▁▁▇
arithmetic Arithmetic 0.63 0.24 0 0.45 0.64 0.82 1 2681 0 ▁▁▃▃▅▇▃▅
word_problem Word Problem 0.38 0.48 0 0 0 1 1 2680 1 ▇▁▁▁▁▁▁▃
sequences Sequences 0.54 0.5 0 0 1 1 1 2662 19 ▇▁▁▁▁▁▁▇
Note:
Summary table shows weighted summary statistics from 4th Grade assessment.

Item Level Summary Statistics

Next, we display summary statistics at the item level for our assessment.

Table 2: Summary Statistics of 4th Grade Assessment Items
Item Label Mean Std Dev Min 25th Percentile Median 75th Percentile Max # Complete Cases # Missing Cases Histogram
Literacy
m8saq4_id 4 Record the number of pictures correctly named. 0.99 0.088 0 1 1 1 1 2681 0 ▁▁▁▁▁▁▁▇
m8saq5_story (6a) Where did Sam and Nakato meet? 0.91 0.29 0 1 1 1 1 2681 0 ▁▁▁▁▁▁▁▇
m8saq6_story (6b) What animal was sleeping next to Nakato? 0.87 0.34 0 1 1 1 1 2681 0 ▂▁▁▁▁▁▁▇
m8saq7_word_choice 7 Choose the correct word from the box and fill in the blank spaces to complete 0.73 0.44 0 0 1 1 1 2681 0 ▃▁▁▁▁▁▁▇
m8saq2_id NA 0.93 0.21 -1 1 1 1 1 2681 0 ▁▁▁▁▁▁▁▇
m8saq3_id NA 0.94 0.22 -1 1 1 1 1 2681 0 ▁▁▁▁▁▁▁▇
Numeracy
m8sbq2_number_sense 2 Please put these numbers in the right order, from lower to higher: 0.79 0.41 0 1 1 1 1 2681 0 ▂▁▁▁▁▁▁▇
m8sbq3a_arithmetic 3a Please provide the correct answers to the following equations: 8+7 0.92 0.27 0 1 1 1 1 2681 0 ▁▁▁▁▁▁▁▇
m8sbq3b_arithmetic 3b Please provide the correct answers to the following equations: 28+27 0.87 0.33 0 1 1 1 1 2681 0 ▁▁▁▁▁▁▁▇
m8sbq3c_arithmetic 3c Please provide the correct answers to the following equations: 335+145 0.87 0.34 0 1 1 1 1 2681 0 ▂▁▁▁▁▁▁▇
m8sbq3d_arithmetic 3d Please provide the correct answers to the following equations: 8-5 0.86 0.34 0 1 1 1 1 2681 0 ▂▁▁▁▁▁▁▇
m8sbq3e_arithmetic 3e Please provide the correct answers to the following equations: 57-49 0.6 0.49 0 0 1 1 1 2681 0 ▆▁▁▁▁▁▁▇
m8sbq3f_arithmetic 3f Please provide the correct answers to the following equations: 7x8 0.71 0.46 0 0 1 1 1 2681 0 ▃▁▁▁▁▁▁▇
m8sbq3g_arithmetic 3g Please provide the correct answers to the following equations: 37x40 0.37 0.48 0 0 0 1 1 2681 0 ▇▁▁▁▁▁▁▃
m8sbq3h_arithmetic 3h Please provide the correct answers to the following equations: 214x104 0.32 0.47 0 0 0 1 1 2681 0 ▇▁▁▁▁▁▁▃
m8sbq3i_arithmetic 3i Please provide the correct answers to the following equations: 6/3 0.68 0.47 0 0 1 1 1 2681 0 ▅▁▁▁▁▁▁▇
m8sbq3j_arithmetic 3j Please provide the correct answers to the following equations: 75/5 0.48 0.5 0 0 0 1 1 2681 0 ▇▁▁▁▁▁▁▆
m8sbq4_arithmetic 4 Which gives smallest answer 0.29 0.45 0 0 0 1 1 2681 0 ▇▁▁▁▁▁▁▂
m8sbq5_word_problem 5 A box contains 26 oranges. How many oranges are contained in 10 boxes? 0.38 0.48 0 0 0 1 1 2680 1 ▇▁▁▁▁▁▁▃
m8sbq6_sequences 6 48 → 24 → 12 → 6 0.54 0.5 0 0 1 1 1 2662 19 ▇▁▁▁▁▁▁▇
m8sbq1_number_sense NA 0.97 0.14 -0.4 1 1 1 1 2681 0 ▁▁▁▁▁▁▁▇
Note:
Summary table shows weighted summary statistics from 4th grade assessment.

Inter-Item Correlations

Next we display inter-item correlations for our 1st grade assessment.

Reliability and Internal Consistency

In Psychometrics, internal consistency is an estimate of test reliability. Cronbach’s Alpha (and other internal consistency coefficients) can take values between 0 and 1, being tests more consistent as the value of this coefficient approaches to 1.

In addition to the internal consistency coefficient for a construct, internal consistency analyses also include item level statistics of internal consistency and item discrimination. You will notice in the output of the alpha() function three additional data frames: Reliability if an item is dropped, Item statistics, and Non missing response frequency for each item. We are interested in the first two data frames.

In the case of the data frame “Reliability if an item is dropped”, we need to check what happens with the overall Cronbach’s alpha reliability when a given item is excluded from the analysis. This analysis allows us to identify inconsistent items when Cronbach’s Alpha increases once a given item is excluded. This information will be shown in the first column of that data frame.

In the case of the data frame “Item Statistics”, we want to identify the item correlation with the construct observed total score when the item is excluded from that total score. Ideally, we want items with a positive correlation between each item and the total score, otherwise that item is probably not measuring the same construct as the others, does not correctly discriminate among examinees, or has a problem in either its content or scoring procedure. From this data frame, we are interested in the fifth column titled “r.drop”.

Internal consistency for literacy items. Cronbach’s alpha = 0.80. Good reliability

For each of our constructs, we report: Cronbach’s alpha for the whole test, Cronbach’s alpha increase or decrease when a given item is removed, and item-total construct score correlation (which is a CTT estimate of item discrimination).

Table 3: Internal Consistency for 4th Grade items
Cronbach alpha for the whole test Cronbach alpha increase or decrease when a given item is removed Item-total construct score correlation
Literacy
m8saq4_id 0.6189186 0.6153251 0.3272989
m8saq5_story 0.6189186 0.5049962 0.5097398
m8saq6_story 0.6189186 0.5352178 0.4389510
m8saq7_word_choice 0.6189186 0.6108375 0.3470873
m8saq2_id 0.6189186 0.5916605 0.3109383
m8saq3_id 0.6189186 0.5749860 0.3656653
Math
m8sbq2_number_sense 0.8095120 0.7995306 0.4075663
m8sbq3a_arithmetic 0.8095120 0.8057075 0.3085536
m8sbq3b_arithmetic 0.8095120 0.8000624 0.4045448
m8sbq3c_arithmetic 0.8095120 0.7971726 0.4494597
m8sbq3d_arithmetic 0.8095120 0.8032319 0.3500001
m8sbq3e_arithmetic 0.8095120 0.7959397 0.4582536
m8sbq3f_arithmetic 0.8095120 0.8025180 0.3730637
m8sbq3g_arithmetic 0.8095120 0.7921426 0.5035299
m8sbq3h_arithmetic 0.8095120 0.7930495 0.4938025
m8sbq3i_arithmetic 0.8095120 0.7875969 0.5574098
m8sbq3j_arithmetic 0.8095120 0.7895035 0.5331731
m8sbq4_arithmetic 0.8095120 0.8044927 0.3387689
m8sbq5_word_problem 0.8095120 0.7902748 0.5263015
m8sbq6_sequences 0.8095120 0.8023162 0.3833062
m8sbq1_number_sense 0.8095120 0.8118666 0.1452517

IRT Parameter Estimates from 2PL Model

## 
Iteration: 1, Log-Lik: -19239.372, Max-Change: 0.58833
Iteration: 2, Log-Lik: -18760.692, Max-Change: 0.35780
Iteration: 3, Log-Lik: -18648.103, Max-Change: 0.22528
Iteration: 4, Log-Lik: -18611.549, Max-Change: 0.14301
Iteration: 5, Log-Lik: -18597.554, Max-Change: 0.09173
Iteration: 6, Log-Lik: -18591.785, Max-Change: 0.05936
Iteration: 7, Log-Lik: -18589.314, Max-Change: 0.03870
Iteration: 8, Log-Lik: -18588.231, Max-Change: 0.02540
Iteration: 9, Log-Lik: -18587.750, Max-Change: 0.01676
Iteration: 10, Log-Lik: -18587.401, Max-Change: 0.00558
Iteration: 11, Log-Lik: -18587.376, Max-Change: 0.00378
Iteration: 12, Log-Lik: -18587.365, Max-Change: 0.00256
Iteration: 13, Log-Lik: -18587.356, Max-Change: 0.00084
Iteration: 14, Log-Lik: -18587.356, Max-Change: 0.00057
Iteration: 15, Log-Lik: -18587.356, Max-Change: 0.00038
Iteration: 16, Log-Lik: -18587.355, Max-Change: 0.00013
Iteration: 17, Log-Lik: -18587.355, Max-Change: 0.00008
## 
## 
## Calculating information matrix...
variable varlabel par_discrimination CI_2.5_discrimination CI_97.5_discrimination par_difficulty CI_2.5_difficulty CI_97.5_difficulty
m8sbq3a_arithmetic 3a Please provide the correct answers to the following equations: 8+7 1.288308 1.1018453 1.474771 -2.3489087 -2.6011293 -2.0966882
m8sbq3b_arithmetic 3b Please provide the correct answers to the following equations: 28+27 1.518569 1.3326330 1.704505 -1.6228306 -1.7712649 -1.4743962
m8sbq3c_arithmetic 3c Please provide the correct answers to the following equations: 335+145 1.744360 1.5378893 1.950830 -1.4160590 -1.5379081 -1.2942099
m8sbq3d_arithmetic 3d Please provide the correct answers to the following equations: 8-5 1.161752 1.0114891 1.312015 -1.7835585 -1.9744571 -1.5926599
m8sbq3e_arithmetic 3e Please provide the correct answers to the following equations: 57-49 1.337480 1.1933274 1.481632 -0.2397872 -0.3182753 -0.1612990
m8sbq3f_arithmetic 3f Please provide the correct answers to the following equations: 7x8 1.060150 0.9342482 1.186051 -0.8528520 -0.9716221 -0.7340818
m8sbq3h_arithmetic 3h Please provide the correct answers to the following equations: 214x104 2.388348 2.1097922 2.666903 0.7329485 0.6636030 0.8022939
m8sbq3i_arithmetic 3i Please provide the correct answers to the following equations: 6/3 2.343537 2.0824928 2.604581 -0.4378229 -0.5018134 -0.3738325
m8sbq3j_arithmetic 3j Please provide the correct answers to the following equations: 75/5 2.123664 1.8984693 2.348860 0.2173163 0.1562017 0.2784309
m8sbq4_arithmetic 4 Which gives smallest answer 1.162880 1.0155712 1.310188 1.2180227 1.0798370 1.3562083
m8sbq5_word_problem 5 A box contains 26 oranges. How many oranges are contained in 10 boxes? 2.315217 2.0596192 2.570816 0.5656756 0.5006870 0.6306642 
## 
Iteration: 1, Log-Lik: -5830.307, Max-Change: 0.88424
Iteration: 2, Log-Lik: -5696.303, Max-Change: 0.47266
Iteration: 3, Log-Lik: -5653.818, Max-Change: 0.35006
Iteration: 4, Log-Lik: -5638.664, Max-Change: 0.26212
Iteration: 5, Log-Lik: -5632.429, Max-Change: 0.19928
Iteration: 6, Log-Lik: -5629.558, Max-Change: 0.15364
Iteration: 7, Log-Lik: -5626.688, Max-Change: 0.05631
Iteration: 8, Log-Lik: -5626.555, Max-Change: 0.04614
Iteration: 9, Log-Lik: -5626.475, Max-Change: 0.03772
Iteration: 10, Log-Lik: -5626.344, Max-Change: 0.00996
Iteration: 11, Log-Lik: -5626.342, Max-Change: 0.00756
Iteration: 12, Log-Lik: -5626.340, Max-Change: 0.00586
Iteration: 13, Log-Lik: -5626.338, Max-Change: 0.00188
Iteration: 14, Log-Lik: -5626.338, Max-Change: 0.00160
Iteration: 15, Log-Lik: -5626.338, Max-Change: 0.00134
Iteration: 16, Log-Lik: -5626.338, Max-Change: 0.00032
Iteration: 17, Log-Lik: -5626.338, Max-Change: 0.00023
Iteration: 18, Log-Lik: -5626.338, Max-Change: 0.00018
Iteration: 19, Log-Lik: -5626.338, Max-Change: 0.00007
## 
## 
## Calculating information matrix...
variable varlabel par_discrimination CI_2.5_discrimination CI_97.5_discrimination par_difficulty CI_2.5_difficulty CI_97.5_difficulty
m8saq4_id 4 Record the number of pictures correctly named. 1.925961 1.5173608 2.334562 -2.7035597 -3.0197952 -2.3873243
m8saq5_story (6a) Where did Sam and Nakato meet? 3.517523 2.6629158 4.372130 -1.3226741 -1.4237018 -1.2216465
m8saq6_story (6b) What animal was sleeping next to Nakato? 1.987024 1.6772915 2.296757 -1.3828586 -1.5079573 -1.2577600
m8saq7_word_choice 7 Choose the correct word from the box and fill in the blank spaces to complete 1.365816 1.1558093 1.575824 -0.8166798 -0.9263449 -0.7070148
m8saq2_id NA 1.043920 0.8697339 1.218106 -1.6805747 -1.9076533 -1.4534961
m8saq3_id NA 1.464549 1.2266772 1.702420 -1.9711770 -2.1869537 -1.7554004

Cut Score for Math

A set of experts from UIS reviewed the SDI 4th grade math test. According to the experts, items 3C, 3F, 3I, 5, and 6 are aligned with the skills expected in 4th grade. Item 3G is aligned to the content of 5th grade. The remaining items are either aligned with skills in earlier grades or are measuring skills not covered in the GPF.

We will take the following approach to assigning a cut score for our math assessment:

## `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'

variable mean sd p0 p25 p50 p75 p100 complete missing hist
cut_score_14 0.33 0.47 0 0 0 1 1 2681 0 ▇▁▁▁▁▁▁▃
m8sbq1_number_sense 0.95 0.22 0 1 1 1 1 2681 0 ▁▁▁▁▁▁▁▇
m8sbq2_number_sense 0.79 0.41 0 1 1 1 1 2681 0 ▂▁▁▁▁▁▁▇
m8sbq3a_arithmetic 0.92 0.27 0 1 1 1 1 2681 0 ▁▁▁▁▁▁▁▇
m8sbq3b_arithmetic 0.87 0.33 0 1 1 1 1 2681 0 ▁▁▁▁▁▁▁▇
m8sbq3c_arithmetic 0.87 0.34 0 1 1 1 1 2681 0 ▂▁▁▁▁▁▁▇
m8sbq3d_arithmetic 0.86 0.34 0 1 1 1 1 2681 0 ▂▁▁▁▁▁▁▇
m8sbq3e_arithmetic 0.6 0.49 0 0 1 1 1 2681 0 ▆▁▁▁▁▁▁▇
m8sbq3f_arithmetic 0.71 0.46 0 0 1 1 1 2681 0 ▃▁▁▁▁▁▁▇
m8sbq3g_arithmetic 0.37 0.48 0 0 0 1 1 2681 0 ▇▁▁▁▁▁▁▃
m8sbq3h_arithmetic 0.32 0.47 0 0 0 1 1 2681 0 ▇▁▁▁▁▁▁▃
m8sbq3i_arithmetic 0.68 0.47 0 0 1 1 1 2681 0 ▅▁▁▁▁▁▁▇
m8sbq3j_arithmetic 0.48 0.5 0 0 0 1 1 2681 0 ▇▁▁▁▁▁▁▆
m8sbq4_arithmetic 0.29 0.45 0 0 0 1 1 2681 0 ▇▁▁▁▁▁▁▂
m8sbq5_word_problem 0.38 0.48 0 0 0 1 1 2680 1 ▇▁▁▁▁▁▁▃
m8sbq6_sequences 0.54 0.5 0 0 1 1 1 2662 19 ▇▁▁▁▁▁▁▇
num_correct 11.53 3.44 0 9 12 14 17 2681 0 ▁▁▂▅▇▇▇▆
school_ipw 209.25 131.52 8.86 101.63 169.88 306.79 544.71 2681 0 ▅▇▃▂▁▁▁▁